Dynamic Global Tone Mapping with Integrated 3D Color Look-Up Table

ABSTRACT

The processing of RGB image data can be optimized by performing optimization operations on the image data when it is converted into the YCbCr color space. For each Y-layer of the YCbCr image data, a 2D LUT is generated. The YCbCr image data is converted into optimized CbCr image data using the 2D LUTs, and optimized YCbCr image data is generated by blending CbCr image data corresponding to multiple Y-layers. The optimized YCbCr image data is converted into sRGB image data, and a tone curve is applied to the sRGB image data to produce optimized sRGB image data.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation and claims priority to and thebenefit of U.S. patent application Ser. No. 15/181,141, filed Jun. 13,2016, the contents of which are hereby incorporated by reference as iffully set forth.

BACKGROUND Field of Art

The disclosure generally relates to the field of digital image and videoprocessing, and more particularly to the implementation of color mappingand tone mapping within a camera architecture.

Description of the Related Art

Digital cameras capture images using an electronic image sensor. Exampleimage sensors use CCD (charge-coupled device) or CMOS (complementarymetal-oxide semiconductor) technology. For example, a CMOS image sensorcontains photodiodes corresponding to pixels of a captured image. Thecaptured image is presented in a RAW color space associated with theCMOS image sensor. The chroma, hue, and lightness of the captured imagecan be adjusted using color mapping after the image is captured by thecamera. Color mapping maps the RAW color space to a target color spacein order to achieve an accurate or preferred color reproduction forcaptured images. The adjustment to the chroma and hue can result in aless than desirable smoothness for various hues with varying chroma.Conventional color mapping methods adjust the chroma and hue using 3Dlook-up tables (“LUTs”) for chroma, hue, and lightness that are storedon the camera.

In addition to color mapping, tone mapping can play an important role inthe capture, storage and reproduction of images because often the bitdepth of the image's pixels has a limited dynamic range that isinadequate to capture all of the light intensities present in real-lifescenes. Accordingly, tone maps specify which ranges of luminance aremapped to each bit value. General tone maps can be applied to imagesregardless of the content of the images. For example, tone map bitvalues may be allocated evenly across the luminance spectrum. However,tone maps can also be adapted based on image content to best representthe real-life scene captured in the image. For example, a scene that isdark or mostly shadowed may use a tone map that allocates more bitvalues to the darker end of the luminance spectrum than a tone map foran average scene. Similarly, a scene that is particularly bright orlight may use a tone map that allocates more bit values to the lighterend of the luminance spectrum than a tone map for an average scene.However, when a scene contains both shadowed and bright features,application of a single tone map may compress features on either edge ofthe spectrum due to the limited bit depth of the tone map.

Accordingly, the optimization of color and tone mapping in such a way asto minimize undesirable artifacts and improve image quality is animportant consideration in the capture of images and the subsequentcolor and tone rendering of the captured images.

SUMMARY

A camera system may be configured to perform particular operations oractions by virtue of having software, firmware, hardware, or acombination of them installed on the camera system that in operationcauses or cause the camera system to perform the actions. One or morecomputer programs may be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. Onegeneral aspect includes a camera system that includes an image signalprocessor (“ISP”). The ISP may be configured to receive YCbCr imagedata. The ISP may generate, for each of a plurality of Y-layers of theYCbCr image data, a 2D color look-up table (“LUT”). The ISP may convertthe YCbCr image data into optimized CbCr image data, for example, usingthe 2D LUTs. The ISP may generate optimized YCbCr image data byblending, for at least two Y-layers, the corresponding CbCr image dataand convert the optimized YCbCr image data into tone-mapped optimizedsRGB image data. The tone-mapped optimized sRGB image data may be storedin a memory of the camera system.

Implementations may include one or more of the following features. TheISP of the camera system may be further configured to convert a raw RGBcolor space into a YCbCr reference color space. The camera system mayalso convert raw RGB image data into the YCbCr image data using theYCbCr reference color space by transforming the raw RGB color space intoan enhanced RGB color space using a linear transformation. Convertingthe raw RGB color space into a YCbCr reference color space may includeconverting the enhanced RGB color space into a non-linear RGB colorspace using a non-linear conversion model. Converting the raw RGB colorspace into a YCbCr reference color space may include transforming thenon-linear RGB color space into the YCbCr reference color space in axvYCC color space using a transformation matrix. Generating a 2D colorLUT may include accessing a set of reference grid points and estimatinga transformation matrix for each reference grid point for differentY-layers. Converting the YCbCr image data into optimized CbCr image datamay include determining optimized CbCr points for each reference gridpoint for the different Y. The camera system may be configured toinclude layers using the estimated transformation matrices. The camerasystem may be configured to transform the YCbCr image data intooptimized CbCr image data using the optimized CbCr points. Transformingthe YCbCr image data into optimized CbCr image data using the optimizedCbCr points may include performing bilinear interpolation on theoptimized CbCr points for different Y-layers. Generating a 2D color LUTmay include accessing a target YCbCr color space and determining anoptimized chroma gain map for different Y-layers based on the YCbCr.Converting the YCbCr image data into optimized CbCr image data mayinclude converting the YCbCr image data into optimized CbCr image datausing the optimized chroma gain maps and the optimized hue shift maps.Converting the optimized YCbCr image data into tone-mapped optimizedsRGB image data may include converting the optimized YCbCr image datainto sRGB image data, and applying a tone curve to the sRGB image datato produce tone-mapped optimized sRGB. The camera system may alsoinclude image data. Implementations of the described techniques mayinclude hardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a method that may include converting, by acamera, light incident upon an image sensor of the camera into raw RGB.The method may also include accessing image data by an ISP of thecamera. The image data may include a raw RGB color space correspondingto the raw RGB image data. The method may also include converting, bythe ISP of the camera, the raw RGB color space into a YCbCr referencecolor space. The method may also include receiving, by an ISP of acamera, YCbCr image data and generating, by the ISP of the camera, foreach of a plurality of Y-layers of the YCbCr image data, a 2D color LUT.The method may include converting, by the ISP of the camera, the YCbCrimage data into optimized CbCr image data using the 2D LUTs andgenerating, by the ISP of the camera, optimized YCbCr image data byblending, for at least two Y-layers, the corresponding CbCr image data.The method may include converting, by the ISP of the camera, theoptimized YCbCr image data into tone-mapped optimized sRGB image dataand storing, by a memory of the camera, the tone-mapped optimized sRGBimage data.

Implementations may include one or more of the following features. Themethod may further include converting, by a camera, light incident uponan image sensor of the camera into raw RGB image data. The method mayinclude accessing, by an ISP of the camera, a raw RGB color spacecorresponding to the raw RGB image data. The method may includeconverting, by the ISP of the camera, the raw RGB color space into aYCbCr reference color space by transforming the raw RGB color space intoan enhanced RGB color space using a linear transformation. Convertingthe raw RGB color space into a YCbCr reference color space may furtherinclude converting the enhanced RGB color space into a non-linear RGBcolor space using a non-linear conversion model. Converting the raw RGBcolor space into a YCbCr reference color space may further includetransforming the non-linear RGB color space into the YCbCr referencecolor space in a xvYCC color space using a transformation matrix.Generating a 2D color LUT may include accessing a set of reference gridpoints and estimating a transformation matrix for each reference gridpoint for different Y-layers. Converting the YCbCr image data intooptimized CbCr image data may include determining optimized CbCr pointsfor each reference grid point for the different Y-layers using theestimated transformation matrices and transforming the YCbCr image datainto optimized CbCr image data using the optimized CbCr points.Transforming the YCbCr image data into optimized CbCr image data usingthe optimized CbCr points may include performing bilinear interpolationon the optimized CbCr points for different Y-layers. Generating a 2Dcolor LUT may include accessing a target YCbCr color space anddetermining an optimized chroma gain map for different Y-layers based onthe YCbCr reference color space and the target YCbCr color space. Themethod may further include determining an optimized hue shift map fordifferent Y-layers. Converting the YCbCr image data into optimized CbCrimage data may include converting the YCbCr image data into optimizedCbCr image data using the optimized chroma gain maps and the optimizedhue shift maps.

BRIEF DESCRIPTION OF DRAWINGS

The disclosed embodiments have other advantages and features which willbe more readily apparent from the detailed description, the appendedclaims, and the accompanying figures (or drawings). A brief introductionof the figures is below.

FIG. 1 illustrates an example high-level block diagram of a camerasystem for implementing color mapping and tone mapping within a digitalimage, according to one embodiment.

FIG. 2 illustrates an example high-level block diagram of a color andtone mapping engine, according to one embodiment.

FIG. 3A illustrates an example color space based on chroma, hue, andlightness, according to one embodiment.

FIG. 3B illustrates a concept diagram of generating 2D look up tables(LUTs) based on the example color space of FIG. 3A, according to oneembodiment.

FIG. 4 illustrates a method for determining transform coefficients for atarget color space based on a standard color space, according to oneembodiment.

FIG. 5 illustrates a process for converting RAW points to optimizedpoints using transform coefficients, according to one embodiment.

FIG. 6 illustrates a process for 3D color mapping an image, according toone embodiment.

FIG. 7 illustrates a process for converting a camera RAW RGB color spaceinto a YCbCr color space, according to one embodiment.

FIG. 8 illustrates a process for generating a 2D color LUT associatedwith optimized CbCr points based on reference grid points, according toa first embodiment.

FIG. 9 illustrates a process for generating a 2D color LUT associatedwith optimized CbCr points based on an optimized chroma gain map and anoptimized hue shift map, according to a second embodiment.

FIG. 10 illustrates a process for generating an optimized sRGB imagefrom blending optimized CbCr points of different Y layers, according toone embodiment.

FIG. 11 illustrates a method for integrating dynamic global tone mappinginto a 3D color LUT, according to one embodiment.

DETAILED DESCRIPTION

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Example Camera Configuration

FIG. 1 illustrates an example high-level block diagram of a camerasystem 100 for implementing color mapping and tone mapping within adigital image, according to one embodiment. The camera 100 of theembodiment of FIG. 1 includes one or more microcontrollers 102, a systemmemory 104, a synchronization interface 106, a controller hub 108, oneor more microphone controllers 110, an image sensor 112, a lens andfocus controller 114, a color and tone mapping engine 116, one or morelenses 120, one or more LED lights 122, one or more buttons 124, one ormore microphones 126, an I/O port interface 128, a display 130, and anexpansion pack interface 132.

The camera 100 includes one or more microcontrollers 102 (such as aprocessor) that control the operation and functionality of the camera100. For instance, the microcontrollers 102 can execute computerinstructions stored on the memory 104 to perform the functionalitydescribed herein. It should be noted that although LUT generation, colorspace conversion, color mapping, and tone mapping are described hereinas performed by the camera 100, in practice, the camera 100 can captureimage data, can provide the image data to an external system (such as acomputer, a mobile phone, or another camera), and the external systemcan generate a color map and a tone map based on the captured imagedata.

A lens and focus controller 114 is configured to control the operation,configuration, and focus of the camera lens 120, for instance based onuser input or based on analysis of captured image data. The image sensor112 is a device capable of electronically capturing light incident onthe image sensor 112 and converting the captured light to image data.The image sensor 112 can be a CMOS sensor, a CCD sensor, or any othersuitable type of image sensor, and can include correspondingtransistors, photodiodes, amplifiers, analog-to-digital converters, andpower supplies.

A system memory 104 is configured to store executable computerinstructions that, when executed by the microcontroller 102, perform thecamera functionalities described herein. The system memory 104 alsostores images captured using the lens 120 and image sensor 112. Thememory 104 can include volatile memory (e.g., random access memory(RAM)), non-volatile memory (e.g., a flash memory), or a combinationthereof.

A synchronization interface 106 is configured to communicatively couplethe camera 100 with external devices, such as a remote control, anothercamera (such as a slave camera or master camera), a computer, or asmartphone. The synchronization interface 106 may transfer informationthrough a network, which allows coupled devices, including the camera100, to exchange data other over local-area or wide-area networks. Thenetwork may contain a combination of wired or wireless technology andmake use of various connection standards and protocols, such as WiFi,IEEE 1394, Ethernet, 802.11, 4G, or Bluetooth.

A controller hub 108 transmits and receives information from user I/Ocomponents. In one embodiment, the controller hub 108 interfaces withthe LED lights 122, the display 130, and the buttons 124. However, thecontroller hub 108 can interface with any conventional user I/Ocomponent or components. For example, the controller hub 108 may sendinformation to other user I/O components, such as a speaker.

A microphone controller 110 receives and captures audio signals from oneor more microphones, such as microphone 126A and microphone 126B.Although the embodiment of FIG. 1 illustrates two microphones, inpractice, the camera can include any number of microphones. Themicrophone controller 110 is configured to control the operation of themicrophones 126. In some embodiments, the microphone controller 110selects which microphones from which audio data is captured. Forinstance, for a camera 100 with multiple microphone pairs, themicrophone controller 110 selects one microphone of the pair to captureaudio data.

The color and tone mapping engine 116 is configured to optimize thecaptured image data obtained from the image sensor 112 based on themapping processes described herein. The mapping processes include colormapping, tone mapping, or any combination thereof. Color mappinggenerates one or more 3D LUTs associated with optimized image data witha preferred color rendering by mapping the captured image data to targetcolor image data. The preferred color rendering describes colorreproduction for use in matching colors of captured image data withcolors of target image data. The colors of the target image data can berepresentative of colors of a real-world image perceived by a human(defined in physical device-independent color space), or of colorsdisplayed by various displays or monitors (defined in device-dependentcolor space). Tone mapping converts the captured image data totone-mapped image data with a preferred tone (e.g., contrast,brightness) rendering. The preferred tone rendering describes tonereproduction for use in matching the limited (or low) dynamic range ofimage data captured by the camera 100 with a high dynamic range (HDR) ofreal-world images, of images perceived by a human, or of imagesdisplayed by various displays or monitors. A dynamic range of real-worldscene is the difference between the highest luminance level in the sceneand the lowest luminance level. In the illustrated embodiment of FIG. 1,the color and tone mapping engine 116 is located within the camera 100.In some embodiments, the color and tone mapping engine 116 is locatedexternal to the camera 100 for instance in a post-processing computersystem, in a cloud server, and the like. In some embodiments, a separatetone mapping engine is located within the camera 100, or external to thecamera 100. The color and tone mapping engine 116 is further explainedin FIG. 2.

Additional components connected to the microcontroller 102 include anI/O port interface 128 and an expansion pack interface 132. The I/O portinterface 128 may facilitate the camera 100 in receiving or transmittingvideo or audio information through an I/O port. Examples of I/O ports orinterfaces include USB ports, HDMI ports, Ethernet ports, audioports,and the like. Furthermore, embodiments of the I/O port interface 128 mayinclude wireless ports that can accommodate wireless connections.Examples of wireless ports include Bluetooth, Wireless USB, Near FieldCommunication (NFC), and the like. The expansion pack interface 132 isconfigured to interface with camera add-ons and removable expansionpacks, such as an extra battery module, a wireless module, and the like.

Color Space Configuration

FIG. 2 illustrates an example high-level block diagram of the color andtone mapping engine, according to one embodiment. In some embodiments,the color and tone mapping engine 116 is implemented within the systemmemory 104 of FIG. 1. The color and tone mapping engine 116 includes acolor point store 205, a preference point store 210, a transformcoefficients store 215, a look up table (LUT) store 220, a transformcoefficient generator 225, a standard transform module 230, a LUTgenerator 235, a 3D color mapping module 240, a tone mapping module 250,and a tone mapping store 252. The color and tone mapping engine 116 caninclude different and/or additional components than those illustrated inthe embodiment of FIG. 2 to perform the functionalities describedherein.

The color point store 205 stores color signal information for variouscolor spaces. For instance, the color point store 205 can store colorsignal information for RGB-type color spaces (e.g., sRGB, Adobe RGB,Adobe Wide Gamut RGB, etc.), CIE defined standard color spaces (e.g.,CIE 1931 XYZ, CIELUV, CIELAB, CIEUVW, etc.), Luma pluschroma/chrominance-based color spaces (e.g., YIQ, YUV, YDbDr, YPbPr,YCbCr, xvYCC, LAB, etc.), hue and saturation-based color spaces (e.g.,HSV, HSL), CMYK-type color spaces, and any other suitable color spaceinformation. In some embodiments, the color point store 205 can storecolor signal information for a RAW RGB color space and one or moreunique color spaces that can be defined by a user.

The stored color signal information for each color space includes one ormore dimensions representing color in the color space. For example, theRGB color space has three dimensions representing color: R for red, Gfor green, and B for blue. For a chroma and hue model, the dimensionsinclude chroma, hue, and lightness. For the YCbCr color space, thedimensions include luma signal (Y), blue-yellow chroma (Cb), andred-green chroma (Cr). The one or more dimensions representing a colorin a color space will be referred to collectively as a “point” herein.Each point can be stored in an association with a color in acorresponding color space in the color point store 205.

The standard transform module 230 applies a transform and gamma curvemanipulation model to RAW points in image data received from the camera100. Data captured by image sensors 112 vary for each image sensor 112on a camera. Thus, the color space of the captured data differs fromimage sensor to image sensor, and the standard transform module 230converts the captured data to a standard color space (such as standardRGB). For example, the standard transform module 230 applies a matrixtransform to the RAW points and converts them. Conventional transformsinclude a matrix transform (e.g., 3×3 matrix transforms, matrixmultiplication transforms, matrix normalization transforms, and thelike), gamma curve manipulation transforms, linear transforms (e.g., CIEXYZ to sRGB, etc.), and any other suitable color space transform fromthe RAW color space to a standard RGB (sRGB) color space.

The preference point store 210 stores preference information for variouspoints (e.g., the points stored in the color point store 205).Preference information for a point can include predefined tuninginformation for the point, for example an increase, decrease, or otherchange in a value associated with one or more dimensions of the point.Preference information can be the same or can differ for one or morepoints of a color space. Preference information can be based on thecolor space, and/or can be user-defined or user-selected.

A target color space can be generated based on preference informationstored in the preference point store 210 applied to one or more“initial” points in an initial color space such as a standard colorspace (e.g., sRGB). For example, if preference information describeschanges to chroma and hue information for an initial point in an initialcolor space, the target color space is generated by applying the changeto chroma and hue information to the initial point in the initial colorspace. For each point in the initial color space, an image datarelationship or association can be identified between the initial pointand a corresponding “target” point in the target color space. The imagedata relationship can be a linear regression, polynomial regression, orany other suitable data fitting describing the relationship between theinitial point of the initial color space and the corresponding targetpoint of the target model.

The transform coefficient generator 225 generates transform coefficients(or simply “coefficients” hereinafter) based on the image datarelationships between initial points in the initial color space andtarget points in the target color space, and stores the generatedcoefficients in the transform coefficients store 215. For example, in apolynomial fit, the coefficients are parameters defining the polynomialfit. Thus, in general, coefficients are parameters defining image datarelationships, such as polynomial fits, between initial points in aninitial color space and target points in a target color space. In oneembodiment, the initial color space is the standard color space asgenerated by the standard transform module 230 from RAW points or datain a RAW color space.

The look up table (LUT) generator 235 generates a LUT for each of one ormore transforms mapping an initial color space to a target color spacebased on the relationships between the initial points and correspondingtarget points for one or more lightness levels. Each generated LUTstores associations between initial points in the initial color spaceand target points in the target color space based on the transformcoefficients and image data relationship associated with the initialpoints in the initial color space and corresponding target points in thetarget color space for at least one lightness level. In one embodiment,the LUT stores chroma and hue information in an initial color space, andcorresponding chroma and/or hue information in a target color space.Thus, the LUT defines optimized points for the chroma and hueinformation for each of one or more initial points within the targetcolor space. The LUTs generated by the LUT generator 235 are stored inthe LUT store 220. In various embodiments, a plurality of LUTs can becombined by the LUT generator 235 to produce a combined LUT, and thecombined LUT can be stored in the LUT store 220. The LUTs can be storedin association with the initial color space, the target color space, orany combination thereof.

The 3D color mapping module 240 generates a 3D LUT associated with anoptimized image by mapping an input color space to a target color spacebased on color space conversion. An input color space is a RAW colorspace associated with the image sensor 112. For example, the input colorspace is a RAW RGB color space characterized by the captured image datafrom the image sensor 112. In some embodiments, the input color spacedefines digital values of input points, e.g., linear 3D color cubepoints, color points obtained from a linear RGB image captured by thecamera 100. The target color space defines digital values of targetpoints for the color mapping process performed by the 3D color mappingmodule 240. In some embodiments, the input color digital values (e.g.,camera RAW RGB) are based on by physical color charts and light sources.The physical color charts are used for color comparisons andmeasurements, e.g., for checking the color reproduction of an imagingdevice. The target color digital values are calculated based on themeasurement of lighting spectral power distribution and spectralreflectance of physical color charts. In some embodiments, the input andthe target color digital values are calculated based on synthesizedspectral reflectance and spectral power distribution of CIE standardlight sources. The optimized color space describes a color space havingcolor points (also referred to as optimized points) which are closer tocorresponding target points in the target color space than correspondinginput points in the input color space.

Color space conversion is used to interconvert the input color space,the target color space, and the optimized color space in a first colorspace into a corresponding input color space, a corresponding targetcolor space, and a corresponding optimized color space, respectively, ina second color space. The color space describes a specific organizationof colors (e.g., a range of colors) for color reproduction. The colorspace can be the RGB color space, the YCbCr color space, or the Labcolor space. For simplicity, the corresponding input color space, thecorresponding target color space, and the corresponding optimized colorspace in the second color space are called an input reference colorspace, a target reference color space, and an optimized color referencemodel, respectively. The first color space is different from the secondcolor space. In one embodiment, the first color space is the RGB space.In such embodiments, the corresponding input color space, target colorspace, and optimized color space are an input RGB color space, a targetRGB color space, and an optimized RGB color space, respectively. In someembodiments, the second color space is an YCbCr space. In suchembodiments, the corresponding input reference color space, targetreference color space, and optimized reference color space are an inputYCbCr reference color space, a target YCbCr reference color space, andan optimized YCbCr reference color space, respectively. Color spaceconversion is further explained below.

The generated 3D LUT can include the input points of the input colorspace, the input reference points of the input reference color space,the optimized points of the optimized color space, the optimizedreference points of the optimized reference color space associationsamong these sets of points (e.g., conversion coefficients, transformcoefficients, data relationship, and dimension associations), or anycombination thereof. The generated 3D LUT is stored in the LUT store220. In some embodiments, color points associated with the color spacesof a 3D color mapping are stored in the color point store 205.

In the embodiment illustrated in FIG. 2, the 3D color mapping module 240includes a color space conversion module 242, a 2D color mapping module244, and a combination module 246. Other embodiments of the 3D colormapping module 240 can include different and/or additional modules otherthan the ones shown in FIG. 2. The 3D color mapping module 240 isfurther described with reference to the description of FIGS. 6-10.

The Color space conversion interconverts the input color space, thetarget color space, and the optimized color space in a first color spaceinto a corresponding input color space, a corresponding target colorspace, and a corresponding optimized color space in a second color spacebased on corresponding image data relationships. For example, the colorspace conversion module 242 can convert the input camera RAW RGB colorspace into the input YCbCr reference color space, can convert the targetRGB color space into the target YCbCr reference color space, and canconvert the optimized YCbCr reference color space into the optimized RGBcolor space. In some embodiments, the image data relationships are basedon one or more linear associations (e.g., linear fitting, interpolation,or regression analysis), more or more nonlinear associations (e.g.,nonlinear fitting, interpolation, or regression analysis), one or moredefined transformation matrix based on standards (e.g., ITU-R BT.601,ITU-R BT.709), or any combination thereof. Image data relationships andassociations among these image data relationships (e.g., conversioncoefficients, transform coefficients, data relationship, and dimensionassociations) are stored in the LUT store 220. The color spaceconversion module 242 is further described in FIG. 7.

The 2D color mapping module 244 generates a 2D color LUT associated with2D optimized points based on mapping the input reference color space tothe target reference color space for one or more values from aparticular color dimension. In one embodiment, both the input and targetreference color spaces have three color dimensions. The 2D color LUT isgenerated based on two color dimensions from the input and targetreference color space with respect to the third color dimension of theinput and target reference color space. For example, given an inputYCbCr reference color space and a target YCbCr reference color space,the 2D color LUT is obtained by mapping the color points associated withCbCr dimensions of the YCbCr reference color space to the correspondingpoints associated with CbCr dimensions of the target YCbCr referencecolor space for one or more values of the Y layer from the input andtarget YCbCr reference color space. Mapping the input reference colorspace to the target reference color space is determined by one or moretransformation matrix defined by the relationship between the input andtarget YCbCr reference color space, one or more dimension associations(e.g., chroma ratio or hue difference) between the input and targetYCbCr reference color spaces, or any combination thereof. Correspondingexamples are described in conjunction with FIG. 6 and FIG. 9.

The combination module 246 generates a 3D color LUT associated with anoptimized image by blending the 2D LUTs associated with the particularcolor dimension from the input and target reference color space. The 3Dcolor LUT is generated based on the 2D LUTs associated with each valueof the particular color dimension. For example, the 3D color LUT isgenerated based on a 2D color LUT based on CbCr dimensions for eachvalue of the Y layer. Thus, the 3D color LUT is representative of the 2Dcolor LUT corresponding to each Y layer. An optimized image is generatedby blending the 2D LUTs with CbCr dimensions corresponding to multiplelevels of a Y layer in the 3D color LUT. To generate the optimized imageof a threshold quality, a threshold number of 2D LUTs can be blended.The combination module 246 is further described in FIG. 10.

The tone mapping module 250 converts input image data to a tone-mappedimage by applying a tone map of a particular bit depth to the inputimage data. In some embodiments, the input image data is the image datacaptured by the image sensor 112. In other embodiments, the input imagedata is obtained from the 3D color LUT, e.g., color-mapped image,optimized data from the 3D color LUT, luma signal intensities associatedwith the color-mapped image data, or any other data associated with the3D color LUT. To select the tone map that best reflects and preservescontent in the input image data, the tone mapping module 250 analyzesthe input image data and computes a tone curve (e.g., a gain curve) thatdescribes a relationship between luma signal values and bit values ofthe input image data. The tone curve is used to generate a tone map thatassigns luma signal ranges or values to bit values (a “bit assignmentscheme” associated with the tone map, as used herein) globally (alsoreferred to as a global tone mapping) or locally (also referred to as alocal tone mapping). A global tone map maps the luminance of every pixelin the input mage data using a global set of bit values. A local tonemap, in contrast, maps the luminance of different pixels in the inputimage data to different sets of bit values based on, for instance, localfeatures of the image data. The tone-mapped image, the correspondingtone map and tone curve, and any associated analysis results can bestored in the tone mapping store 252 for future use. In someembodiments, the bit assignment scheme is stored in association with thetone mapped image.

In some embodiments, a tone map can be applied to an image before acolor map, while in other embodiments, the tone map can be applied afterthe color mapping (as described in conjunction with FIG. 10). In someembodiments, the tone mapping is integrated with the color mapping toobtain an image with preferred color and tone mapping in one step (asdescribed in conjunction with FIG. 11). For example, a tone curve isgenerated based on the input image data associated with 3D color LUT. Insuch case, the tone curve changes with luminance intensities and scenetype of the input image. Such tone curve is called a dynamic tone curve.The tone mapping for generating a dynamic tone curve is called a dynamictone mapping. By integrating the dynamic tone mapping and color mapping,a toned, colored-mapped image is generated. The example is furtherdescribed in conjunction with FIG. 11.

In some embodiments, the tone mapping module 250 also includes a gammacorrection module (not shown in FIG. 2). The gamma correction module isused to encode or decode the input image data based on a nonlinear model(e.g., a gamma function) associated with a corresponding encoding ordecoding gamma parameter. Typically, a gamma value that is smaller than1 is called an “encoding gamma value”, and a gamma value that is largerthan 1 is called a “decoding gamma value”. The gamma correction moduledetermines a gamma curve associated with the gamma function andcorresponding gamma parameter based on linearity (or non-linearity)properties of the input image data. For example, if the input image datais linear RGB image data, by applying the gamma function with anencoding gamma value to the linear RGB image data, the resulting outputimage data becomes nonlinear. The output image data can become linearagain after a subsequent application of the gamma function with adecoding gamma value.

Example Color Space

FIG. 3A illustrates an example color space 300 based on chroma, hue, andlightness, according to one embodiment. A chroma, hue, and lightnesscolor space is illustrated for the purposes of discussion, but it shouldbe noted that any other suitable color space can be used with theprocesses described herein. Chroma 310 represents saturation, hue 305represents color, and lightness 315 represents the lightness of thechroma 310 and hue 305. For varying lightness measures 315A and 315B,various points 320A and 320B representing chroma are determined forcorresponding hue levels 305. As shown in FIG. 3A, various lightness inthe color space 300 are represented on various z-axis planes of thecolor space 300, also shown in FIG. 3B. Adjusting a chroma measure from320A to 320B can be difficult depending on a corresponding hue measure.For example, adjusting chroma 310 for a red hue 305 varies saturation ofred more noticeably than adjusting chroma 310 for a green hue 305.

FIG. 3B illustrates the generation of 2D look up tables (LUTs) based onthe example color space of FIG. 3A, according to one embodiment. Acombined LUT 330 is generated for various LUTs 325, each associated witha lightness level 315. Following the example of a chroma and hue colorspace 300, LUTs 325A and 325B are generated for lightness levels 315A,315B. A plurality of the LUTs 325 are combined (e.g., by addition,subtraction, average, or any other suitable processing combination ofmappings from points in the initial color space to points in the targetcolor space) to generate one or more combined LUTs 330. For example,LUTs 325A and 325B can be generated for lightness 315A (representativeof 100% light) and lightness 315B (representative of 75% light),respectively. A combined LUT 330 can be generated by averaging, for eachpoint in an initial color space, the point of the target color space inLUT 325A and LUT 325B. The combined LUT maps a point in the initialcolor space to a point in the target color space representative of anaverage of all points in the target color space mapped to the point inthe initial color space by the LUTs 325. The process of generating LUTs325 for various lightness 315 levels of the chroma and hue color space300 and combining a plurality of the generated LUTs 325 to generate oneor more combined LUTs 330 is further described below in FIGS. 4 and 5.

Determining Transform Coefficients from a Standard Color Space to aTarget Color Space

FIG. 4 illustrates a method for determining transform coefficients for atarget color space based on a standard color space, according to oneembodiment. In other embodiments, the method includes different,additional, or fewer steps than those depicted by FIG. 4. Additionally,in some embodiments, the steps described in conjunction with FIG. 4 maybe performed in different orders.

A set of points are accessed 405 within a standard color space and,herein, these points of the standard color space are referred to as“standard points.” As described previously in FIG. 2, a point includescolor space dimensions (e.g., a point in the RGB color space includesdimensions R, G, B; a point in a chroma hue model includes dimensionschroma, hue, and lightness).

Dimensions of each standard point in the set of standard points aredetermined 410. If the standard RGB color space is used, then chroma andhue dimensions can be determined based on R, G, and B values associatedwith the standard points. For example, the R, G, and B values associatedwith the standard points can be converted to chroma and hue dimensionsusing RGB to HSV or HSL conversions, conversion charts, or any othersuitable conversion method. In alternative methods, the dimensions ofeach standard point can be the R, G, B values themselves, lightnessdimensions, or any other suitable dimension of various color spaces.

Preference information is received 415 for tuning the dimensions of theset of standard points. Preference information for tuning dimensions ofa point can be predefined tuning information for the point such asincreasing, decreasing, or otherwise altering one or more of thedimensions of the point. Preference information can bedimension-specific. Following the previous example, chroma and huedimensions can be tuned using preference information specifying how toadjust one or more of the dimensions in the initial color space.Preference information can specify adjustments in, for instance, huecolor, chroma, or both hue color and chroma. Preference information canbe stored in the preference point store 210.

A target point is determined 420 for each standard point in the set ofstandard points based on the preference information. The preferenceinformation is applied to one or more of the standard points in the setof standard points to determine 420 a corresponding target point in atarget color space. For example, the preference information can specifyan increase in hue and a decrease in chroma. The determined target pointcan be stored in the color point store 205 and can be stored inassociation with the target color space, the standard color space, orany combination thereof.

One or more transform coefficients are determined 425 for each standardpoint in the set of standard points based on the preference information.Transform coefficients are determined 425 based on a relationshipbetween each target point and a corresponding standard point in the setof standard points. For example, the transform coefficient generator 225applies a mathematical model to determine a relationship between eachtarget and corresponding standard point, and the transform coefficientsare coefficients representative of the mathematical model. The transformcoefficients are stored 430, for example, in the transform coefficientsstore 215. The transform coefficients can also be stored in associationwith the associated mathematical model.

Determining Transform Coefficients from a Standard Color Space to aTarget Color Space

FIG. 5 illustrates a process for converting RAW points to optimizedpoints using transform coefficients, according to one embodiment. Inother embodiments, the method includes different, additional, or fewersteps than those depicted by FIG. 5. Additionally, in some embodiments,the steps described in conjunction with FIG. 5 may be performed indifferent orders.

A set of RAW points (each RAW point comprising a point in a respectiveRAW color space) are accessed 505. For example, the RAW points are RGBpoints within an image from the camera 100. The RAW points can be storedin the color point store 205 in association with the respective RAWcolor space.

The set of RAW points are converted 510 to a set of standard points. Thestandard points are points in the standard color space as previouslydescribed in FIG. 4. The set of RAW points are converted using, forexample, transforms mapping RAW points from the camera in a RAW RGBcolor space to points in a standard RGB (sRGB) color space.

Stored transform coefficients are accessed 515. The stored transformcoefficients are variables of a polynomial fit from the target model tothe optimized color space. The set of standard points are converted 520to a set of optimized points using the transform coefficients in acorresponding mathematical model such as a polynomial fit. Thus, thecoefficients are parameters of the polynomial fit. As a result of theconversion, corresponding optimized points are determined. The optimizedpoints are stored in the color point store 205, for example, and can beassociated with the target color space. In another embodiment, theoptimized points can be stored in association with an optimized colorspace.

In an alternative embodiment, the transform coefficients accessed fromthe transform coefficients store 215 are for points not included in theset of standard points to convert 520. In this embodiment, themathematical model applied to a point can be weighted based on nearbypoints associated with the transform coefficients. In one embodiment,the mathematical model is a polynomial fitting defined by:

x _(a) =p _(x0) +p _(x1) x+p _(x2) y+p _(x3) x ² +p _(x4) xy+p _(x5) y ²

y _(a) =p _(y0) +p _(y1) x+p _(y2) y+p _(y3) x ² +p _(y4) xy+p _(y5) y²  (1)

This and other suitable mathematical models for use in the systemdescribed herein are described in more detail in “Image registration bylocal approximation methods”, Image and Vision Computing, Vol. 6, 1988,pp. 255-261, the contents of which are incorporated herein by referencesin their entirety.

For polynomial coefficients p_(x0), p_(x1), p_(x2), p_(x3), p_(x4),p_(x5), p_(y0), p_(y1), p_(y2), p_(y3), p_(y0), and p_(y5), where (x, y)is the original points in the warping and (x_(a), y_(a)) is the secondorder polynomial of each original point, weighted averages of equation(1) for nearby points (x₁, y₁) and (x₂, y₂) can be used to determine(x₃, y₃). For example, if distance between points (x₁, y₁) and (x₃, y₃)is shorter than the distance between points (x₂, y₂) and (x₃, y₃), thenthe polynomial coefficients associated with points (x₁, y₁) are given ahigher weight than the polynomial coefficients associated with points(x₂, y₂). The weight can be proportional to the distance between thepoints and weighted by the greater distance or any other value. Forexample, the weighted optimized point (x_(p),y_(p)) can be determined bythe following:

$\begin{matrix}{\left( {x_{p},y_{p}} \right) = \frac{\sum\limits_{i = 1}^{N}\left\{ {{W\left( R_{i} \right)} \times \left( {x_{ai},y_{ai}} \right)} \right\}}{\sum\limits_{i = 1}^{N}\left\{ {W\left( R_{i} \right)} \right\}}} & (2)\end{matrix}$

In equation 2, (x_(ai), y_(ai)) is the polynomial of each original pointbased on equation 1, N is the number of standard points that are usedfor local weighted mean averages of the mathematical equation (e.g.,polynomial fits), and these standard points have a weighting function W(R_(i)) larger than 0. W (R_(i)) can be determined as:

$\begin{matrix}{{W\left( R_{i} \right)} = \left\{ {{\begin{matrix}{{W\left( R_{i} \right)},} & {0 \leq R \leq 1} \\{0,} & {R > 1}\end{matrix}R_{i}} = \frac{\sqrt{\left( {x_{t} - x} \right)^{2} + \left( {y_{t} - y} \right)^{2}}}{R_{M}}} \right.} & (3)\end{matrix}$

In equation 3, (x, y) is a standard point to be converted 520, (x_(t),y_(t)) is a standard point, M is the number of neighboring points of (x,y), and R_(M) is the distance of (x, y) from its Mth nearest originalpoints. For example, W (R_(i)) is a monotonically decreasing continuousfunction of R_(i), which can be user-defined.

An additional set of optimized points are determined 525 based on theset of optimized points. For example, the additional set of optimizedpoints is determined 525 based on weighted averages of nearby pointsfrom the set of optimized points. A similar weighting of mathematicalmodels applied to nearby points can be used to determine the additionalset of optimized points.

The set of optimized points are combined 530 with the additional sets ofoptimized points and a LUT is generated 535 storing association ofpoints in the set of standard points to corresponding points in the setof optimized points and additional set of optimized points. In oneembodiment, the generated LUT corresponds to a lightness level of thecolor space 300. The generated LUT is stored 540. For example, thegenerated LUT is stored 540 in the LUT store 220.

The generated LUTs stored 540 in the LUT store 220 can be combined asdescribed above to generate one or more combined LUTs that representvarious lightness levels. For example, a first generated LUT can be for100% lightness and a second generated LUT can be for 50% lightness.These two LUTs can be combined to generate a combined LUT. The two LUTscan be combined by adding, subtracting, normalizing, weighting each LUT,or any combination thereof and stored in the LUT store 220. The combinedLUT can then be applied, or can be stored for subsequent use.

Generating and Implementing Color Mapping and Tuning Based on a TargetModel

FIG. 6 illustrates a process 600 for 3D color mapping an image,according to one embodiment. The process 600 can be implemented by the3D color mapping module 240. In other embodiments, the process includesdifferent, additional, or fewer steps than those depicted by FIG. 6.Additionally, in some embodiments, the steps described in conjunctionwith FIG. 6 may be performed in different orders.

The 3D color mapping module 240 converts 610 a RAW RGB color spacereceived from a camera (“camera RAW RGB color space”) into an YCbCrreference color space. For example, the 3D color mapping module 240accesses the color point store 205 for obtaining a set of camera RAW RGBcolor space points 602 (“camera RAW RGB points”). The 3D color mappingmodule 240 converts 610 the RAW RGB points 602 associated with the RAWRGB color space into corresponding YCbCr points 604 associated with theYCbCr reference color space based on a relationship betweencorresponding color spaces In some embodiments, the 3D color mappingmodule 240 converts target points associated with a target RGB colorspace corresponding to the camera RAW RGB color points 602 intocorresponding target YCbCr points 605 associated with a target YCbCrreference color space. The color space conversion is further explainedin conjunction with FIG. 7.

The 3D color mapping module 240 generates 620 a 2D color LUT associatedwith a 2D optimized CbCr points 606 by mapping the input YCbCr points604 associated with the input YCbCr reference color space to the targetYCbCr points 605 associated with the target YCbCr reference color spacefor different levels of the Y layer of the YCbCr color space from theinput YCbCr reference color space and target YCbCr reference colorspace. In some embodiments, the optimized CbCr points 606 are determinedby the reference grid points 603, which describe intermediate points forconverting the input color points (e.g., camera RAW RGB points 602) intooptimized points (e.g., optimized CbCr points 606). The reference gridpoints 603 are evenly distributed in the coordinate system of CbCr, asfurther described in FIG. 8. In some embodiments, the optimized CbCrpoints 606 are determined by a relationship between features (e.g.,chroma ratio, hue difference) of the input and target YCbCr referencecolor space, as further described in FIG. 9.

The 3D color mapping module 240 blends 630 the 2D optimized CrCb points606 of at least two levels of Y layer to generate an optimized YCbCrimage 608 associated with a 3D color LUT. In some embodiments, a tonemapping is applied after the 3D color LUT for obtaining a tone-mappedoptimized image, as further explained in FIG. 10.

FIG. 7 illustrates a process 610 for converting a camera RAW RGB colorspace into an YCbCr reference color space, according to one embodiment.In other embodiments, the process of FIG. 7 includes different,additional, or fewer steps than those depicted by FIG. 7. Additionally,in some embodiments, the steps described in conjunction with FIG. 7 maybe performed in different orders.

The color space conversion module 242 transforms 720 the camera RAW RGBcolor space 602 into an enhanced RGB color space 702 using a lineartransformation. All the color points of the RAW RGB color space 602 withvalues higher than 1 or lower than 0 are thus retained in the enhancedRGB color points 702 without clipping after performing the lineartransformation. In some embodiments, the linear transformation caninclude a matrix transform (e.g., 3×3 matrix transforms, matrixmultiplication transforms, matrix normalization transforms, and thelike), gamma curve manipulation transforms, and any other suitable colorspace transform from the camera RAW color space to an enhanced RGB colorspace.

The color space conversion module 242 converts 740 the enhanced RGBcolor space 702 into a non-linear RGB color space 704 using a non-linearconversion. For example, the non-linear conversion can include theconversion expressed by equation (4):

$\begin{matrix}\left\{ \begin{matrix}{{{If}\mspace{14mu} R},G,{B \leq {- 0.018}},} & \begin{matrix}{R^{\prime} = {{{- 1.009} \times \left( {- R} \right)^{0.45}} + 0.099}} \\{G^{\prime} = {{{- 1.009} \times \left( {- G} \right)^{0.45}} + 0.099}} \\{B^{\prime} = {{{- 1.009} \times \left( {- B} \right)^{0.45}} + 0.099}}\end{matrix} \\{{{{If} - 0.018} < R},G,{B < 0.018},} & \begin{matrix}{R^{\prime} = {4.5 \times R}} \\{G^{\prime} = {4.5 \times G}} \\{B^{\prime} = {4.5 \times B}}\end{matrix} \\{{{If}\mspace{14mu} R},G,{B \geq 0.018},} & \begin{matrix}{R^{\prime} = {{1.009 \times (R)^{0.45}} - 0.099}} \\{G^{\prime} = {{1.009 \times (G)^{0.45}} - 0.099}} \\{B^{\prime} = {{1.009 \times (B)^{0.45}} - 0.099}}\end{matrix}\end{matrix} \right. & (4)\end{matrix}$

In equation 4, R, G, and B are enhanced RGB points 702 after lineartransformation from camera RAW RGB color space; R′, G′ and B′ arenon-linear RGB points 704 after conversion using Eq. (4).

The color space conversion module 242 transforms 760 the non-linear RGBcolor space 704 into an YCbCr reference color space 604 in an xvYCCspace using a transformation matrix based on any suitable color spacestandard (e.g., ITU-R BT.601, ITU-R BT.709). The xvYCC space isdescribed in more detail in “Colour management—Extended-gamut YCC colourspace for video applications—xvYCC,” IEC 61966-2-4: Multimedia systemsand equipment—Colour measurement and management, 2006, the contents ofwhich are incorporated herein in their entirety. For example, thetransformation matrix can be the transformation matrix expressed byequations (5)-(6):

$\begin{matrix}{\begin{bmatrix}Y^{\prime} \\{Cb}^{\prime} \\{Cr}^{\prime}\end{bmatrix}_{xvYCC} = {\begin{bmatrix}{m\; 11} & {m\; 12} & {m\; 13} \\{m\; 21} & {m\; 22} & {m\; 23} \\{m\; 31} & {m\; 32} & {m\; 33}\end{bmatrix}\begin{bmatrix}R^{\prime} \\G^{\prime} \\B^{\prime}\end{bmatrix}}} & (5) \\{{Y_{xvYCC} = {{round}\left( {\left( {{219 \times Y_{xvYCC}^{\prime}} + 16} \right) \times 2^{n - 8}} \right)}}{{Cb}_{xvYCC} = {{round}\left( {\left( {{224 \times {YCb}_{xvYCC}^{\prime}} + 128} \right) \times 2^{n - 8}} \right)}}{{Cr}_{xvYCC} = {{round}\left( {\left( {{224 \times {YCr}_{xvYCC}^{\prime}} + 128} \right) \times 2^{n - 8}} \right)}}} & (6)\end{matrix}$

In equations 5 and 6, m11-m33 is a 3×3 transformation matrix used toconvert R′, G′, B′ to Y′, Cb′, Cr′ in an xvYCC space and defined inITU-R BT.601 protocol or the ITU-R BT.709 protocol, both are which areincorporated herein in their entirety. The Y, Cb, Cr values are YCbCrcolor points 604, which are digital quantization values of YCbCr in thexvYCC space converted by equation 5. The Y, Cb, Cr values have the rangefrom 2^(n-8) to 254×2^(n-8), wherein n is the bit-depth of signalvalues, which can be larger than or equal to 8.

Similarly, the target YCbCr points 605 can be generated by equations 4through 6 by replacing the enhanced RGB color space associated with thecamera RAW RGB color space with an enhanced RGB color space associatedwith the target RGB color space. In some embodiments, the color spaceconversion module 242 converts an optimized YCbCr reference color spaceinto a corresponding RGB color space using an inverse transformationderived from equations 4 through 6. Thus, the color space conversionmodule 242 can interconvert any RGB color space into a correspondingYCbCr color space during color mapping.

FIG. 8 illustrates a process 620A for generating a 2D color LUTassociated with optimized CbCr points based on reference grid points830, according to a first embodiment. In other embodiments, the processincludes different, additional, or fewer steps than those depicted byFIG. 8. Additionally, in some embodiments, the steps described inconjunction with FIG. 8 may be performed in different orders.

The 2D color mapping module 244 accesses 840 input and target CbCrpoints for each reference grid point for different Y layers. In oneembodiment, the input CbCr points are obtained from the input YCbCrreference color space 810 corresponding to a value of the Y layer. Thetarget CbCr points are obtained from the target YCbCr reference colorspace 820 corresponding to the input YCbCr reference color space 810. Tofind the input and target CbCr points for a particular reference gridpoint, the input and target CbCr points are selected within a boundarycentered around the reference grid point. The size of the boundary isdetermined by the particular reference grid point located in the centerof the small area. To avoid any smoothing issue, the number of referencegrid points in CbCr space is equal to or exceeds a threshold value.

The 2D color mapping module 244 estimates 850 a 2×2 transformationmatrix for each reference grid point for different Y layers. Forexample, a 2×2 transformation matrix is estimated based on thecoordinates of the input and target CbCr points within a correspondingboundary for one Y layer. The 2×2 transformation matrix for one Y layercan be defined by equation 7:

$\begin{matrix}{\begin{bmatrix}{Cb}_{{Tgt}\; 1} & \ldots & {Cb}_{TgtN} \\{Cr}_{{Tgt}\; 1} & \ldots & {Cr}_{TgtN}\end{bmatrix} = {\begin{bmatrix}{k\; 11} & {k\; 12} \\{k\; 21} & {k\; 22}\end{bmatrix}_{2 \times 2} \times \begin{bmatrix}{Cb}_{{Init}\; 1} & \ldots & {Cb}_{InitN} \\{Cr}_{{Init}\; 1} & \ldots & {Cr}_{InitN}\end{bmatrix}}} & (7)\end{matrix}$

In the embodiment of equation 7, Cb_(TgtN) and Cr_(TgtN) are the CbCrcoordinates of the Nth target CbCr point; Cb_(IniN) and Cr_(InitN) arethe CbCr coordinates of the Nth input CbCr point; and [k₁₁, k₁₂, k₂₁,k₂₂] are transform coefficients of the 2×2 transformation matrix.

The 2D color mapping module 244 calculates 860 optimized CbCr points foreach reference grid point for different Y layers. For example, the 2×2transformation matrix is applied to the reference grid point located inthe center of a particular small area for determining the optimized CbCrpoints. The optimized CrCb points for one Y layer can be determinedusing equation 8:

$\begin{matrix}{\begin{bmatrix}{Cb}_{OptM} \\{Cr}_{OptM}\end{bmatrix} = {\begin{bmatrix}{k\; 11} & {k\; 12} \\{k\; 21} & {k\; 22}\end{bmatrix}_{2 \times 2} \times \begin{bmatrix}{Cb}_{GridM} \\{Cr}_{GridM}\end{bmatrix}}} & (8)\end{matrix}$

In the embodiment of equation 8, Cb_(OptM) and Cr_(OptM) are optimizedCbCr coordinates of the Mth reference grid point; and Cb_(GridM) andCr_(GridM) are CbCr coordinates of the Mth reference grid point.

The 2D color mapping module 244 transforms 880 the input CbCr pointsfrom the input YCbCr image 870 to corresponding optimized CbCr 890points based on all of the reference grid points 830 and thecorresponding optimized CbCr points 862 using a bilinear interpolationfor different Y layers. In one embodiment, the input YCbCr image 870corresponds to an input YCbCr reference color space in a whole YCbCrcolor space.

FIG. 9 illustrates a process 620B for generating a 2D color LUTassociated with optimized CbCr points based on an optimized chroma gainmap and an optimized hue shift map, according to a second embodiment. Inother embodiments, the process includes different, additional, or fewersteps than those depicted by FIG. 9. Additionally, in some embodiments,the steps described in conjunction with FIG. 9 may be performed indifferent orders.

The 2D color mapping module 244 calculates 910 an optimized chroma gainmap based on the chroma ratio of the input YCbCr reference color space810 and the target YCbCr reference color space 820 for different Ylayers. For example, for each chroma level, the chroma gain of CbCrpoints are determined by the chroma ratio of the input and target CbCrpoints for one Y layer. A relationship between the chroma gain and thehue angle of the set of CbCr points is calculated and interpolated usingspline interpolation for one Y layer.

The 2D color mapping module 244 calculates 920 an optimized hue shiftmap based on the hue difference between the input YCbCr color space 810and the target YCbCr color space 820 for different Y layers. Forexample, for each chroma level, the hue shifts of CbCr points aredetermined by the hue difference between the input and target CbCrpoints. A relationship between the hue shift and hue angle of the set ofCbCr points is calculated and interpolated using spline interpolationfor one Y layer. With the control of different chroma levels, the low,middle and high chroma colors can be adjusted separately.

The 2D color mapping module 244 converts 940 the input YCbCr image 930to optimized CbCr points 950 using an optimized chroma gain map and hueshift map for different Y layers. To avoid smoothing issues, the numberof sample points at different hue angles and chroma levels can be equalto or exceed a predetermined threshold value.

FIG. 10 illustrates a process 630 for generating an optimized sRGB imagefrom blending optimized CbCr points of different Y layers, according toone embodiment. In other embodiments, the process includes different,additional, or fewer steps than those depicted by FIG. 10. Additionally,in some embodiments, the steps described in conjunction with FIG. 10 maybe performed in different orders.

The combination module 246 blends 1020 the optimized CrCb points 1010 ofat least two levels of Y layer for generating an optimized YCbCr image1030 associated with a 3D color LUT. For example, the YCbCr optimizedimage 1030 is generated by combining the optimized CbCr points atdifferent Y layers. In some embodiments, the combination module 246converts 1040 the optimized YCbCr image 1030 into a correspondingoptimized sRGB image 1050. In one embodiment, the optimized sRGB imagecan be obtained by applying an inverse transformation derived fromequations 4 through 6 to convert the optimized YCbCr image to obtain thecorresponding optimized sRGB image.

In some embodiments, the tone mapping module 250 applies 1060 a tonecurve to the optimized sRGB image 1050 for generating the tone-mappedoptimized sRGB image 1070. For example, the applied tone curve can be again curve, and a global tone mapping can be applied to the optimizedsRGB image 1050. In such embodiments, applying a global tone mapping tothe optimized sRGB image 1050 includes applying the gain curve to eachpixel of the optimized sRGB 1050 to get a tone-mapped optimized sRGB1070.

Dynamic Global Tone Mapping with a 3D Color Look-Up Table

In some embodiments, tone mapping is integrated with the color mappingto obtain an image with preferred color and tone mapping in onecomputational process. For example, by integrating dynamic tone mappingand color mapping, a toned, colored-mapped image can be generated.

FIG. 11 illustrates a process for integrating a dynamic global tonemapping into the 3D color LUT, according to one embodiment. The processcan be performed by the color and tone mapping engine 116. Moreparticularly, the process can be performed by the combination of 3Dcolor mapping module 240 and tone mapping module 250. In otherembodiments, the method includes different, additional, or fewer stepsthan those depicted by FIG. 11. Additionally, in some embodiments, thesteps described in conjunction with FIG. 11 may be performed indifferent orders. As shown in FIG. 11, the method includes three steps.Each step is described below.

Step 1 builds the mapping between an input color space (e.g., linear 3Dcolor cube points 1110) and a target color space. In one embodiment, thecolor and tone mapping engine 116 applies 1112 a first 3D color LUT witha fixed gamma curve to linear 3D color cube points 1110 for generatingnonlinear color mapped color points 1113. The fixed gamma curve can bepre-generated and accessed upon the determination to combine tonemapping and color mapping. In some embodiments, the first 3D color LUTis extracted from the LUT store 220, the linear 3D color cube points1110 are extracted from the color point store 205, and the gamma curveis extracted from the tone mapping store 252. Second, the color and tonemapping engine 116 applies 1114 a decoding gamma curve to the nonlinearcolor mapped color points 1113 to generate a color mapped linear 3Dcolor LUT 1116. Third, the color and tone mapping engine 116 calculates1117 luminous intensity of each color point in the linear 3D color LUT1116. For example, the luminous intensity is a weighted function ofcolor mapped linear R, G and B values.

Step 2 generates a dynamic gain curve using a linear 3D color LUT andcorresponding luminance intensities. In one embodiment, the color andtone mapping engine 116 uses a camera linear RGB image as an inputimage. The camera linear RGB image is an image captured using digitalcamera with high resolution. In some embodiments, as shown in FIG. 11,the camera linear RGB image is downscaled to a low-resolution image 1120to reduce processing computation. The steps 1122, 1124, are 1127 aresimilar to the steps 1112, 1114, and 1117, respectively. Instead ofusing linear 3D color cube points 1110 as an input in Step 1, thedownscaled linear RGB image is used as an input for Step 2. After step1127, the luminous intensity 1128 of each color in the downscaled linear3D color LUT 1126 is calculated. The color and tone mapping engineapplies 1129 a global tone mapping to the color mapped linear RGB image1126 and to the calculated luminous intensity 1128 to generate a gaincurve 1130. The gain curve 1130 corresponds to the captured imageassociated with the luminance intensities. The gain curve 1130 changeswith changes in the luminance intensities and in the scene type of inputRGB image; accordingly, the gain curve 1130 can be dynamically updated.The global tone mapping is used to apply the same gain curve to all thepixels in the specific image. In some embodiments, a local tone mappingis used to generate a pixel-wise gain curve by considering pixelneighborhood information for each pixel in the image. The local tonemapping applies the pixel-wise gain curve to each pixel in the image togenerate high contrast image.

Step 3 integrates a dynamic tone mapping with a 3D color LUT. In oneembodiment, the color and tone mapping engine 116 applies 1132 the tonemapping gain curve 1130 to the color mapped linear 3D Color LUT 1116based on the calculated luminous intensity of each color in a linear 3DColor LUT 1118 to generate a tone-mapped color-mapped linear 3D ColorLUT 1134. The color and tone mapping engine 116 applies 1136 theencoding gamma curve used in step 1112 to the tone-mapped color-mappedlinear 3D color LUT 1134 to generate a tone-mapped color-mappednonlinear 3D color LUT 1138. In one embodiment, the color and tonemapping engine 116 applies the tone-mapped color-mapped nonlinear 3Dcolor LUT 1138 to a camera linear RGB image to generate an output imagewith preferred color and tone mapping. In another embodiment, the colorand tone mapping engine 116 applies a second 3D color LUT to thetone-mapped color mapped nonlinear 3D color LUT to generate a secondtone-mapped color-mapped nonlinear 3D color LUT. The color and tonemapping engine 116 applies 1122 the second tone-mapped color-mappednonlinear 3D color LUT to the camera linear RGB image to obtain anoutput image with preferred color and tone mapping.

In some embodiments, the first 3D color LUT and the second 3D color LUTare different. For example, the first 3D color LUT is generated by theLUT generator 235 based on the transform coefficients and image datarelationship associated with the initial points in the initial colorspace and corresponding target points in the target color space. Theinitial and target color space are in the color space, e.g., the sRGBspace. Continuing with this example, the second 3D color LUT can begenerated by the 3D color mapping module 240 by mapping an input colorspace to a target color space based on color space conversion. The inputcolor space is obtained from image data captured by the camera 110,e.g., the camera RAW RGB color space. The input color space can bedifferent from the initial color space used to generate the first 3Dcolor LUT. The input and target color spaces in the first color spaceare converted to corresponding input and target reference color spacesin a second color space. In some embodiments, the first 3D color LUT andthe second 3D color LUT can be the same 3D color LUT.

Additional Configuration Considerations

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms, for example, as illustrated inFIGS. 2 and 3. Modules may constitute either software modules (e.g.,code embodied on a machine-readable medium or in a transmission signal)or hardware modules. A hardware module is tangible unit capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.Further, unless expressly stated to the contrary, “or” refers to aninclusive or and not to an exclusive or.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for asystem and a process for synchronizing multiple image sensors throughthe disclosed principles herein. Thus, while particular embodiments andapplications have been illustrated and described, it is to be understoodthat the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various apparentmodifications, changes and variations may be made in the arrangement,operation and details of the method and apparatus disclosed hereinwithout departing from the spirit and scope defined in the appendedclaims.

What is claimed is:
 1. A camera system, comprising: an image signalprocessor (“ISP”) configured to: receive a YCbCr image data; generate,for each of a plurality of Y-layers of the YCbCr image data, a 2D colorlook-up table (“LUT”); convert the YCbCr image data into optimized CbCrimage data using the 2D LUTs; generate optimized YCbCr image data byblending, for at least two Y-layers, the corresponding CbCr image data;convert the optimized YCbCr image data into tone-mapped optimized sRGBimage data; and store the tone-mapped optimized sRGB image data in amemory of the camera system.
 2. The camera system of claim 1, whereinthe ISP is further configured to: convert a raw RGB color space into aYCbCr reference color space; and convert raw RGB image data into theYCbCr image data using the YCbCr reference color space by transformingthe raw RGB color space into an enhanced RGB color space using a lineartransformation.
 3. The camera system of claim 2, wherein converting theraw RGB color space into a YCbCr reference color space furthercomprises: converting the enhanced RGB color space into a non-linear RGBcolor space using a non-linear conversion model.
 4. The camera system ofclaim 3, wherein converting the raw RGB color space into a YCbCrreference color space further comprises: transforming the non-linear RGBcolor space into the YCbCr reference color space in a xvYCC color spaceusing a transformation matrix.
 5. The camera system of claim 1, whereingenerating a 2D color LUT comprises: accessing a set of reference gridpoints; and estimating a transformation matrix for each reference gridpoint for different Y-layers.
 6. The camera system of claim 5, whereinconverting the YCbCr image data into optimized CbCr image datacomprises: determining optimized CbCr points for each reference gridpoint for the different P-layers using the estimated transformationmatrices; and transforming the YCbCr image data into optimized CbCrimage data using the optimized CbCr points.
 7. The camera system ofclaim 6, wherein transforming the YCbCr image data into optimized CbCrimage data using the optimized CbCr points comprises performing bilinearinterpolation on the optimized CbCr points for different Y-layers. 8.The camera system of claim 1, wherein generating a 2D color LUTcomprises: accessing a target YCbCr color space; determining anoptimized chroma gain map for different Y-layers based on the YCbCrreference color space and the target YCbCr color space; and determiningan optimized hue shift map for different Y-layers.
 9. The camera systemof claim 8, wherein converting the YCbCr image data into optimized CbCrimage data comprises: converting the YCbCr image data into optimizedCbCr image data using the optimized chroma gain maps and the optimizedhue shift maps.
 10. The camera system of claim 1, wherein converting theoptimized YCbCr image data into tone-mapped optimized sRGB image datacomprises: converting the optimized YCbCr image data into sRGB imagedata; and applying a tone curve to the sRGB image data to producetone-mapped optimized sRGB image data.
 11. A method, comprising:converting, by a camera, light incident upon an image sensor of thecamera into raw RGB image data; accessing, by an image signal processor(“ISP”) of the camera, a raw RGB color space corresponding to the rawRGB image data; converting, by the ISP of the camera, the raw RGB colorspace into a YCbCr reference color space; receiving, by an image signalprocessor (“ISP”) of a camera, YCbCr image data; generating, by the ISPof the camera, for each of a plurality of Y-layers of the YCbCr imagedata, a 2D color look-up table (“LUT”); converting, by the ISP of thecamera, the YCbCr image data into optimized CbCr image data using the 2DLUTs; generating, by the ISP of the camera, optimized YCbCr image databy blending, for at least two Y-layers, the corresponding CbCr imagedata; converting, by the ISP of the camera, the optimized YCbCr imagedata into tone-mapped optimized sRGB image data; and storing, by amemory of the camera, the tone-mapped optimized sRGB image data.
 12. Themethod of claim 11, further comprising: converting, by a camera, lightincident upon an image sensor of the camera into raw RGB image data;accessing, by an image signal processor (“ISP”) of the camera, a raw RGBcolor space corresponding to the raw RGB image data; and converting, bythe ISP of the camera, the raw RGB color space into a YCbCr referencecolor space by transforming the raw RGB color space into an enhanced RGBcolor space using a linear transformation.
 13. The method of claim 12,wherein converting the raw RGB color space into a YCbCr reference colorspace further comprises: converting the enhanced RGB color space into anon-linear RGB color space using a non-linear conversion model.
 14. Themethod of claim 13, wherein converting the raw RGB color space into aYCbCr reference color space further comprises: transforming thenon-linear RGB color space into the YCbCr reference color space in axvYCC color space using a transformation matrix.
 15. The method of claim11, wherein generating a 2D color LUT comprises: accessing a set ofreference grid points; and estimating a transformation matrix for eachreference grid point for different Y-layers.
 16. The method of claim 15,wherein converting the YCbCr image data into optimized CbCr image datacomprises: determining optimized CbCr points for each reference gridpoint for the different P-layers using the estimated transformationmatrices; and transforming the YCbCr image data into optimized CbCrimage data using the optimized CbCr points.
 17. The method of claim 16,wherein transforming the YCbCr image data into optimized CbCr image datausing the optimized CbCr points comprises performing bilinearinterpolation on the optimized CbCr points for different Y-layers. 18.The method of claim 11, wherein generating a 2D color LUT comprises:accessing a target YCbCr color space; determining an optimized chromagain map for different Y-layers based on the YCbCr reference color spaceand the target YCbCr color space; and determining an optimized hue shiftmap for different Y-layers.
 19. The method of claim 18, whereinconverting the YCbCr image data into optimized CbCr image datacomprises: converting the YCbCr image data into optimized CbCr imagedata using the optimized chroma gain maps and the optimized hue shiftmaps.
 20. The method of claim 11, wherein converting the optimized YCbCrimage data into tone-mapped optimized sRGB image data comprises:converting the optimized YCbCr image data into sRGB image data; andapplying a tone curve to the sRGB image data to produce tone-mappedoptimized sRGB image data.